Method and system for generating a holographic image having simulated physical properties

ABSTRACT

A method, system and program for producing an interactive three-dimensional holographic image, including the steps of generating, by one or more processors of a computer system, a virtual object and assigning physical properties to the virtual object using metadata. Signals are received from a virtual tool to determine a position of the virtual tool. Interactive force between the virtual tool and the virtual object are calculated based on the signals from the virtual tool and the position of the virtual tool. A modified virtual object is generated based on the interactive forces and the physical properties, and the modified virtual object is displayed as a holographic image. The system may also determine a force feedback according to the position of the virtual tool in relation to the virtual object, send the force feedback to a user through a haptic interface device; and update the force feedback according to movement of the virtual tool in real space.

FIELD OF THE INVENTION

A system and method for controlling and operating a display apparatus,and more particularly, for providing an interactive three-dimensionalholographic display system, method and device for holograms attributedwith simulated physical properties for physical analysis.

BACKGROUND OF THE INVENTION

Holographic displays are used to display objects in three dimensions.Typical interactive devices are incapable of providing an interactivethree-dimensional holographic display that sufficiently displays theresult of interactions with the object being displayed.

SUMMARY OF THE INVENTION

The present invention is directed to solving issues relating to one ormore of the problems presented in the prior art, as well as providingadditional features that will become readily apparent by reference tothe following detailed description when taken in conjunction with theaccompanying drawings.

A method, system and program is provided for producing an interactivethree-dimensional holographic image, including the steps of generating,by one or more processors of a computer system, a virtual object andassigning physical properties to the virtual object using metadata.Signals are received from a virtual tool to determine a position of thevirtual tool. Interactive force between the virtual tool and the virtualobject are calculated based on the signals from the virtual tool and theposition of the virtual tool. A modified virtual object is generatedbased on the interactive forces and the physical properties, and themodified virtual object is displayed as a holographic image. The systemmay also determine a force feedback according to the position of thevirtual tool in relation to the virtual object, send the force feedbackto a user through a haptic interface device; and update the forcefeedback according to movement of the virtual tool in real space.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart illustrating an exemplary process for generatingan object having physical properties and applying forces to the object,in accordance with embodiments of the present invention.

FIG. 2 is a flowchart illustrating a system, in accordance withembodiments of the invention.

FIG. 3 illustrates a functional block diagram of a system including ahaptic interface device, modeling apparatus, and graphics display inaccordance with embodiments of the invention.

FIG. 4 illustrates a view of a virtual environment including a virtualobject and a virtual tool, in accordance with embodiments of theinvention.

FIG. 5 illustrates a view of a virtual environment where a virtual toolcontacts a virtual surface of the virtual object in connection with thehaptic interface location with the virtual object, in accordance withembodiments of the invention.

FIG. 6 illustrates a flowchart of the modification process occurringbetween a virtual tool and a virtual object, in accordance withembodiments of the invention.

FIG. 7 illustrates an exemplary display apparatus configured to producean interactive three-dimensional holographic image, in accordance withembodiments of the invention.

FIG. 8 illustrates exemplary parameters for an exemplary displayapparatus configured to produce an interactive three-dimensionalholographic image, in accordance with embodiments of the invention.

FIG. 9 illustrates an exemplary feedback system of an exemplary displayapparatus, in accordance with embodiments of the present invention.

FIG. 10 illustrates various exemplary interactions with a holographicdisplayed three-dimensional image, in accordance with embodiments of thepresent invention.

FIG. 11 shows an exemplary detailed 3D rendering engine, in accordancewith embodiments of the present invention.

FIG. 12 illustrates an exemplary computing system including aninteractive three-dimensional holographic display, in accordance withembodiments of the present invention.

FIG. 13 illustrates an exemplary digital media player having aninteractive three-dimensional holographic display, in accordance withembodiments of the present invention.

FIG. 14 illustrates an exemplary personal computer having an interactivethree-dimensional holographic display, in accordance with embodiments ofthe present invention.

FIG. 15 illustrates an exemplary mobile telephone having an interactivethree-dimensional holographic display, in accordance with embodiments ofthe present invention.

FIG. 16 illustrate a virtual object in the form of clay displayed as ahologram with a mobile device used to input data such as scale andapplied force with rotation being imparted tot the object by a user, inaccordance with embodiments of the present invention.

FIG. 17 illustrate the virtual object of FIG. 16 after modificationbased on user applied forces and rotation upon the clay object displayedas a hologram, in accordance with embodiments of the present invention.

FIG. 18 illustrates a computer system used for implementing the methodsof the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In the following description of embodiments, reference is made to theaccompanying drawings which form a part hereof, and in which it is shownby way of illustration specific embodiments in which the invention canbe practiced. It is to be understood that other embodiments can be usedand structural changes can be made without departing from the scope ofthe disclosed embodiments.

This invention relates to a display system configured to produce aninteractive three-dimensional holographic image where the digital objectwill be assigned appropriate physical properties as metadata, forexample, hardness, water content, compressive strength, viscosity etc. Acamera and proximity sensor installed in the apparatus may analyze handand/or finger movement patterns around the holographic object andaccordingly will identify a type of force being applied, amount ofapplied force, and impact of applied force upon the holographic object.The display system will calculate the resultant physical reaction of theobject to the forces being applied and display the object taking intoaccount any changes in the physical appearance of the object.

According to an embodiment, the apparatus of the invention may changethe structure and shape of the holographic object based on identifiedforce parameters, material content of the object, and the user-selectedscaling information. The shape of the object may change according to thelaws of the physical object.

Another embodiment is directed to method for producing an interactivethree-dimensional holographic image where software has analyzed theamount of applied force, the direction of the force, and the type ofapplied force, accordingly from a selected scale. The software willrelate the material properties of the object to calculate a resultantshape and the holographic projector will recreate a new shape of theobject gradually based on the applied forces and the physical propertiesof the object. In this way, the user will be able to determine how theobject is changing based on the applied force and associated physicalparameters.

Many types of interactive devices are available for performingoperations in a computing system. Interactive display screens (e.g.,touch screens, in particular) are becoming increasingly popular becauseof their ease and versatility of operation as well as their decliningprice. Touch screens generally allow a user to perform various functionsby touching (e.g., physical contact or near-field proximity) the touchsensor panel using a finger, stylus or other interactive object at alocation dictated by a user interface (UI) being displayed by thedisplay device. Typical touch screens, however, provide atwo-dimensional display on a substantially flat surface.

FIG. 1 is a flowchart illustrating an exemplary process for generatingan object having physical properties and applying forces to the object,in accordance with embodiments of the present invention. Data describingthe object to be modeled, load data, and load path data are firstobtained (block 4). A material and associated material properties arethen selected from a plurality of materials in a material library (block6). The objects and materials may be contained in an object and/ormaterials library or may be created using known computer-aided draftingtechniques. The object shape, load, and load path data are then input tothe selected model to generate object output (block 8). Finally, theobject output is displayed using a graphical format (block 10) as ahologram.

With this general example embodiment having now been presented, a moredetailed embodiment may be described. Accordingly, reference is made toFIG. 2 illustrating a second example embodiment of the presentinvention. In particular, FIG. 2 is a flowchart useful for illustratingan example computer program system and method 20 of the invention. Theembodiment 20 comprises a graphic user interface (block 22) or “GUI” forinterfacing with a user for receiving input data, displaying outputdata, and the like. Those knowledgeable in the art will appreciate thatfor the purposes of inputting data the GUI 22 may comprise any of avariety of forms, with an example comprising visually displayed symbols,characters, and the like on a screen or like display.

Through the GUI (block 22) the embodiment 20 may obtain input data(block 24) to be used for object modeling such as a description of theobject, load data, load path data, material property data, initial stateof stress, state variables, loading increment, and the like. This datamay be input by the user through manual data entry, through automatedentry with menu selection items, by signals received from sensors, etc.In addition to being obtained through the GUI 22, some or all of thisdata may also be obtained through retrieval of existing data stored in adatabase or repository, on a portable medium, or on another like storagemedium.

Once the input data is obtained, it will be communicated to a modelingmodule (dashed line block 26). The modeling module (block 26) includes:selecting a model (block 28) from a model library (block 29), applyingthe model by inputting the data to the model to generate model output(block 30), and probing the model output (block 32).

The model library (block 29) preferably comprises a plurality of models,with at least one of the models comprising one or more constitutivemodels. Also, the model library may be accessible to users through theGUI or other means so that models may be added, modified, and/or deletedas desired over time. The availability of a plurality of models in onelocation for convenient application to the problem at hand is anadvantage of the present invention. In a sense, the invention embodimentcomprising a plurality of models to experiment with thereby provides aworkbench for convenient use of different modeling tools. The model maybe uniquely generated by the user using known computer aided draftingsoftware, and then added to the model library.

A general state of loading can be input manually or linked to outputfrom an analysis performed using a finite element, finite difference,boundary element, meshless technique, or similar method. For example, aninvention program product embodiment may comprise performing a finiteelement analysis on a body, and selecting an individual element of thatbody using a cursor or other selector at the conclusion of the analysis.The load path data associated with this element under the finite elementanalysis will then be used in applying the selected constitutive model.

The embodiment 20 further comprises a step of probing the model (block34). As used herein, the term “probing the model” is intended to broadlyrefer to checking the model output data for performance. For example,the probing of the model may comprise verifying that the model asapplied to the input data obeys basic laws of conservation in mechanics.In addition, the probing of a model may probe the predictive capabilityof the constitutive relation as well as its completeness. For example,the probing of a model may comprise verifying that energy or work is notextracted from a constitutive relation, and thus provides an indicationof the stability of the model.

After applying and probing the model, model output data is obtained(block 36). Generally, the model output data comprises the results ofcalculations and representations made to the input data using theselected model. In summary, the model output generally comprises anupdated state of stress and state variables. The model output data mayalso comprise flags or other logical data that define the yield,failure, and potential surfaces.

The embodiment 20 next comprises steps of mapping the model output data(block 38). Mapping may generally be thought of as providing a frameworkand bounds for visually representing the model output data. Duringmapping, the model output data is further processed for visual displayin a graphical format. In the embodiment 20, mapping is providedaccording to an advanced (block 40) or a direct (block 42) scheme.

Steps of direct mapping (block 42) enable the mapping of selected stressand strain scalar values in two or three dimensions. Direct mapping isused to represent up to three components of stress, and can superimposethe constitutive model geometry over the stress paths in conventionalspace. Direct mapping can map: stress and strain response due to loadingspecified in the loading path module the yield surface, flow rule, andMohr circle of stress in a two dimensional space, any pair of the 12independent components of stress and strain. Other components can alsobe mapped such as step number, constitutive model state variables, andmean-deviatoric stress components in three-dimensional space, anycombination for the stress tensor and principal stress values, which canbe mapped in principal stress space. This allows the stress path to bemapped in relation to the hydrostatic axis and n-line with superimposedconstitutive model geometry.

Direct mapping for constitutive model output data, stress paths, and thelike may be useful for simple loading cases such as uniaxial compressionor extension or triaxial loading. However, under a general loadingcondition, or when examining the stress history result at a point from afinite element analysis, all stress and strain components carry relevantinformation about the loading history experienced. Advanced mapping(block 40) may be useful for such cases.

Advanced mapping techniques may comprise use of a glyph, a hyperstreamline, or other geometric shape to graphically represent secondorder symmetric stress tensors or similar representation techniques.Four or greater stress, strain, or state variables may be representedusing advanced mapping. As an example, in these techniques problemsassociated with visualizing six independent components are resolved byrepresenting three orthogonal unit vectors whose magnitude is equal tothe eigenvalue of the second order tensor, and whose direction isdefined by the corresponding eigenvector. Advanced mapping techniquessuch as use of glyph and glyph-like geometric shapes allow for data thatdescribes six components to be represented using only a threedimensional shape. This may be accomplished by taking advantage ofmapping and rendering densities, surfaces, and like details of theshape. An invention embodiment may comprise using advanced or directmapping based on a user input selection. For example, the GUI 22 mayprovide an interface for a user to select one or more modes of mapping.

Following mapping, the embodiment 20 comprises rendering a graphicaldisplay of the mapped output data (block 44) using the exemplary systemdescribed below. Method and program product embodiments of the inventioncomprise rendering a static graphical display (block 46) or a dynamicdisplay (block 48) of the model output data. The display may bepresented in two or three dimensions, and may comprise animation. Thestatic display component preferably comprises three dimensions,including, for example, a volume and a surface represented by ahologram. In the invention embodiment 20, the steps of rendering adynamic graphical display (block 48) of the model comprise changing thegraphical display in response to changes in the input or other data.

In this manner, a user may quickly and decisively examine the results ofchanges in load and/or load path data, for example. The display may bepresented in a two or three dimensional rendering of loading paths andthe model output data. For three-dimensional renderings, dynamic displayallows the user to navigate through the space to view the model and viewit from different angles as the stress path changes. Dynamic displayalso allows rotation and lateral movement within the display space.

The inventive embodiment 20 further comprises an object or model library50 for use in rendering of a graphic display of the output data. Theobject or model library 50 may comprise data for a variety of geometricshapes that may be used in models such as constitutive models, forinstance. Shapes may include, but are not limited to, three dimensionalshapes such as an arrow, cone, cube, sphere, cylinder, and the like. Ithas been discovered that use of pre-defined and pre-existing shapes froma library speeds the processing of embodiments of the present invention.The final output graphical display may be presented for user viewingusing the GUI 22.

FIG. 3 illustrates a functional block diagram of a system including ahaptic interface device, modeling apparatus, and graphics display inaccordance with embodiments of the invention. In accordance with oneembodiment, a user of the system may use the haptic interface device 100to interact with the virtual object 126 (see FIG. 3) receiving forcefeedback produced by the haptic rendering process and viewing graphicsrendered by the graphics process on a graphic display 44. The hapticinterface device 100 may comprises a user-operated sensor system such asglove embedded with sensors. For example, user may employ a wearabledevice (e.g. ring) on the thumb and/or index finger or an embeddabledevice (e.g., an e-tattoo, or conductive paint) on the nails and/orfingers, so that the device will measure the relative movement amongeach detectable element and also measure the speed and direction ofmovement. In this manner, the paired computing devices will preciselymeasure the movement and forces applied to the virtual object in a veryprecise manner.

In a haptic interface embodiment, the process, as referred to in FIG. 3,is a software process executing on a hardware microprocessor. All theprocesses may execute on one microprocessor, or in other embodiments,one or more processes may execute on different microprocessors, whichare linked by buses, cables, local networks, wide area networks, orglobal computer networks, such as the Internet.

The modeling application as viewed in FIG. 3 as a software applicationexecuting on one computer system. In another embodiment, the modelingapplication executes on one or more computer systems connected by acommunications device, such as a bus, cable, or network connection. Inan alternate embodiment, the modeling application is a hardware device,such as an ASIC (application specific integrated circuit), and one ormore processes of the application are implemented on one or more ASICdevices. In a further embodiment, the modeling application isimplemented as one or more objects, which may execute on one or morecomputer systems.

As shown in FIG. 3, the modeling application 112 is preferably but notrequired to include a haptic rendering process 116, an interactionprocess 118, a modification process 120, and a graphics process 122. Inone embodiment, the functions of the modeling application 112 areimplemented by a different number of processes. In one embodiment, themodeling application 112 includes the haptic rendering process 116 andthe graphics process 122.

The invention may be implemented using an object-oriented approach. Thehaptic rendering process 116 and other processes are implemented assoftware objects. In another embodiment, the virtual object 126 and thevirtual tool 128 (FIG. 4) are implemented as software objects andperform one or more of the functions of the haptic rendering process116. The final image is displayed on the graphics display 114.

The modeling application may a computer program stored on a computerreadable storage media, such as a CD disc, diskette, tape, or othermedia. In another embodiment, the modeling application is a computerprogram distributed over a computer-readable propagated signal, such asa program distributed over the Internet.

As mentioned, the system may include a haptic interface system, as shownin FIG. 3, including the haptic interface device 110 and the hapticrendering process 116 which generates a virtual object of the virtualenvironment to be “touched” and determines the results of theinteraction (discussed in more detail below). The haptic interfacedevice 110 is a tactile or force-feedback device which provides thetouch sensations of interacting with virtual objects 126 to a user ofthe system. Some haptic interface devices 110 consist of anelectro-mechanical linkage which can exert a controllable force on auser's hand. As used herein, “haptic rendering” refers to the creationof a virtual environment with which a user can interact through thesense of touch. The term “haptic rendering process” 116 refers to thecomputer program which generates the haptic aspects of the virtualenvironment and determines the forces to be applied to a user through ahaptic interface. The haptic rendering process 116 generates hapticrepresentations of virtual objects in the virtual environment.

FIG. 4 shows a haptic virtual environment including a virtual object 126and a virtual tool 128. The virtual object 126 of the embodiment shownin FIG. 4 is depicted as a 3-D (three dimensional) block of materialtypically “floating” in the virtual space of the virtual environment.The virtual object 126 has virtual surfaces 125 that represent the“skin” of the virtual object 126. The virtual tool 128 is represented inFIG. 4 as a sphere 134 with a rod or “handle” 132 connected to it. Asmentioned, however, in the embodiment the virtual tool 128 may be auser's hand having appropriate sensors applied thereto as will bedescribed in greater detail below.

In the exemplary illustration of FIGS. 4 and 5, the user uses a hapticinterface device 110 in real space to grasp or manipulate the handle 132of the virtual tool 128 in virtual space. In one embodiment, thelocation of this handle with respect to the virtual tool 128 can bechanged interactively by the user. As used herein, a “haptic virtualenvironment” refers to a computer-generated virtual environment that canbe explored by a user through the sense of touch. In one embodiment, thehaptic virtual environment contains a virtual object 126 that is modelof a real world object that a user is creating in the virtualenvironment. In another embodiment, the haptic virtual environmentincorporates two or more virtual objects 126 that are linked to eachother, such as in a hierarchical arrangement. It should be understoodthat the interaction and/or modification methods described herein may bereadily extended to apply to two or more virtual objects 126 linked orassociated in a haptic virtual environment.

FIG. 5 illustrates a view of a virtual environment where a virtual toolcontacts a virtual surface of the virtual object in connection with thehaptic interface location with the virtual object, in accordance withembodiments of the invention. More specifically, FIG. 5 illustrates avirtual tool 128 contacting the virtual surface 125 of a virtual object126. The user guides the virtual tool 128 using the haptic interfacedevice, represented, in this embodiment, by a stylus 133 in FIG. 5. Theposition and orientation of the tip of the stylus 133 indicate thehaptic interface location 198. Note that, although the user may bemanipulating a literal stylus in some embodiments, the haptic interfacelocation 198 could be controlled by a user interacting with any numberof differently shaped elements such as a finger, thimble, a yoke, or aball. The tip of the virtual stylus 133 is indicated by the hapticinterface location 198. In one embodiment, the haptic rendering process116 tracks the haptic interface location 198, but does not otherwisetrack the shape or location of the entire haptic interface device 110.

The haptic rendering process 116 attempts to move the virtual tool 128so that the origin 127 of the virtual tool 128 matches the hapticinterface location 198. However, unless the haptic rendering process 116is using the virtual tool 128 to remove material from the virtual object26, then the haptic rendering process 116 typically does not allow thevirtual tool 128 to penetrate the virtual object 126. Thus, as shown inFIG. 5, the user has attempted to move the virtual tool 128 into thevirtual object 126, which is indicated by the haptic interface location198 within the virtual object 126. The haptic rendering process 116calculates a resistance to the movement of the virtual tool 128 into thevirtual object 126 based on the material properties of the virtualobject 126 entered as data by the user. This calculation is based, forexample, on a connection 129 between the tool origin 127 and the hapticinterface location 198. In another embodiment, the connection 129includes a virtual spring 131 to enable calculation of the forces in aknown mathematical manner. In one embodiment, the connection 129includes a virtual dash-pot. Thus, if the user attempts to move thevirtual tool 128 further into the virtual object 126, the hapticrendering process 116 calculates an increasing resistance force that isfed back to the user through the haptic interface device 110 based onthe virtual spring 131. While the force calculation methods of FIG. 5utilize a spring or dash-pot system, other known methods of calculatingforces will be known to those of skill in the art and available throughknown computer drafting and machining software.

As already described, the user interacts with the virtual object 126 inthe virtual environment through a virtual tool 128. The user may selectany shape for the tool 128. The shape of the tool 128, along with othercharacteristics, such as interaction mode, determines the interactionwith the virtual object 126. In one embodiment, the tool 128 may berepresented as a series of discrete points in virtual space whichoutline a three-dimensional shape of the tool 128. The virtual tool 128is modeled as a set of discrete points for the purposes of hapticinteraction and collision detection with the virtual object 26. Inanother embodiment, the points of the virtual tool 128 are created by analgebraic equation or any other continuous or piecewise mathematicalmethod suitable for determining a 3-D shape in a virtual environment. Inanother embodiment, the tool 128 can be represented directly bycontinuous or piecewise mathematical equations, rather than by discretepoints. The virtual tool 128 may take on any of a number of shapes thatmay be useful for a user when using a virtual tool 128 to create avirtual object 126 in the virtual environment. Typical shapes mayinclude a sphere or cylinder. In another embodiment, the user selectsone or more interaction modes for the virtual tool 128, such as a sandpaper mode, which causes the tool 128 to induce friction of the virtualobject 126 or to remove material gradually from the virtual object 126,much like using real sandpaper to smooth the shape of a block of wood inthe real world.

FIG. 6 illustrates a flowchart of the modification process occurringbetween a virtual tool and a virtual object, in accordance withembodiments of the invention. More specifically, FIG. 6 illustrates aflowchart of the modification process occurring between a virtual tool128 and virtual object 126. First, a virtual object 126 is generated atstep 390 and physical properties are assigned to the virtual object 126at step 395 using metadata. Next, a virtual tool 128 is determined orgenerated in virtual space that represents the haptic interface device110 that the user is manipulating in real space (step 400). The virtualtool 128 may comprises a computer generated tool or may comprise auser-operated input such as a glove having sensors mounted thereto. Forexample, the virtual tool 128 may be a wearable device (e.g. ring) onthe thumb and index finger or an embeddable device (e-tattoo, orconductive paint) on the nails or fingers, so the device will measurethe relative movement between the tool 128 and object 126 and also therelative speed and direction of movement. In one embodiment, the hapticrendering process 116 generates the virtual tool 128. In anotherembodiment, the system detects the user-operated input by receivingsignals indicative of motion and forces applied by the tool 128. Theuser then selects a modification mode that determines what kind ofmodification occurs to the virtual object 126 as a result of interactionbetween the virtual object 126 and the virtual tool 128 (step 402). Themodification modes can include application of force, application oftemperature, application of spin, material modification, and othermaterial modification modes. The material modification mode can includespinning, smoothing, mirroring, and other material modification modes.

In step 404, sensors determine the location of a user or user-operatedinput in real space. In one embodiment the user is manipulating a hapticinterface device 110 such as a glove and sensors determine the positionof the haptic interface device 110 in real space.

The modeling application 112 then determines the location of thediscrete points of the virtual tool 128 relative to the location of thevirtual object 126 (step 406). In one embodiment the haptic renderingprocess 116 determines these locations. The haptic rendering process 116then calculates an interaction force between the virtual tool 128 andthe virtual object 126 based on the locations of the points of thevirtual tool 28 and the location of the virtual object 126 (step 408).In an embodiment, the user feels the interaction force through thehaptic interface device 110, which thus provides feed back to the useron the interaction of the virtual tool 128 with the virtual object 126.In one embodiment, the haptic rendering processor 116 provides theinteraction force to the haptic interface device 110. The virtual object126 may include a virtual surface 125 and the position and orientationof the virtual tool 128 is determined relative to the virtual surface125 based on the locations of the points of the virtual tool 28 comparedto the virtual surface 125.

The modeling application 112 then produces a modified virtual object 126by modifying the virtual object 126 based on the modification mode, theposition of the virtual tool 128, the physical properties of the virtualobject 126, and the location of the virtual object 126 (step 410). Themodification processor 120 produces the modified virtual object 126. Forexample, if the virtual tool 128 is in a translation mode and the useris attempting to move the virtual object 126 with the virtual tool 128,then the modification processor 120 calculates the forces applied to thevirtual object 126 as the user pushes the tool 128 against the object126. If the modification mode is a spinning mode, and the user isapplying a tangential force to the virtual object 126 with the virtualtool 128 (as though spinning a basketball), then the modificationprocessor 120 calculates the spinning motion of the virtual object 126based on the force and amount of tangential force that the user isapplying to the virtual object 26.

The modified virtual object 126 is then output from the system. In oneembodiment, the output is a modified visual image of the virtual object126 that is output to a graphics display 114 by the modeling application112 or graphics processor 122. In one embodiment, the output alsoincludes a new or modified shape, which the user feels through thehaptic device. The user then decides whether to continue with themodification process (step 414). If the user decides to continue in thesame modification mode, the user makes an additional movement of thevirtual tool 128, and the haptic rendering process 116 determines thenew position of the virtual tool 128 (step 406). The user may decide toselect a different modification mode (step 416) and returns to step 402to select the new modification mode. If the user does not decide to makeany further modifications, then the virtual object 126 may be displayed,output, or saved to a disk, tape, or other data storage device (step418). Output may include output or export to on an alternate file formator a printing device or a device that provides a physical, real worldmodel of the virtual object 126.

According to one aspect of the present invention, the calculation ofinteractive forces at step 408 may be scaled by the system to achievecertain reactions that cannot easily be achieved by actual physicalcontact. For example, the user may wish to spin the virtual object 126by applying a tangential force; however, the user's finger having asensor applied thereto my not be able to achieve the desired spin. Inthis case, the applied force by the user may be scaled mathematically,e.g. by multiplying the forces applied to the virtual object 126, toachieve a greater spin. In this example, the user may apply a tangentialforce of 10N and the system will scale the tangential force by a factorof 4 to be 40N. Other types of scaling is also envisioned with thisinvention to increase or decrease the applied force or other outsidefactor acting on the virtual object 126.

In accordance with this invention, a display apparatus is configured toproduce an interactive three-dimensional holographic image where thephysical properties of the object being displayed are taken into accountduring the rendering of the object. A coherent light source can produceone or more beams, based on obtained image data of an object to display.A lens assembly can be configured to direct the one or more beams, bydynamically changing a deflection angle, to form a holographic image ofthe object based on a focal length of the lens and a location of anobserver. Further, one or more optical sensors can be configured toobtain information regarding whether an interactive device interruptsthe one or more beams, in order to determine a location of theinteractive device (e.g., a user's finger) with respect to theholographic image, based on the obtained information from the one ormore optical sensors.

A holographic image can be created, for example, with two parabolicmirrors that are facing each other. A 3D object to be imaged can belocated in the center of the lower mirror assembly and the object can beprojected through an opening in the top mirror assembly. In essence themirror assembly can allow imaging of the object at a virtually infinitenumber of views, each at different viewing angles, creating aholographic image of the 3D object above the top mirror assembly.

The display apparatus, according to embodiments described herein cancreate the same holographic image by projecting a plurality of objectviews, each at different viewing angles, above a lens assembly. Acomputer rendering engine, for example, can render a plurality of objectviews at different viewing angles based on a virtual object.Accordingly, a truly unobtrusive three-dimensional holographic displaycan be provided, without the need of a reflective medium. Moreover, auser can interact with the holographic image, based on informationobtained from the optical sensors receiving reflected light from aninteractive device interrupting a beam forming the image.

FIG. 7 illustrates an embodiment of a display apparatus configured toproduce an interactive three-dimensional holographic image. As shown inFIG. 7, a lens assembly 100 includes a bottom lens 125 that can be acollimating lens, which can redirect a beam 153 from an XY scanner 152into a direction that is perpendicular to the bottom surface of the toplens 120, for example. It is noted that XY scanner 152 can projectmonochrome or multi-color beam(s) 153. The top lens 120 can then deflectthe beam 153 into a direction that is dependent upon the surfacefunction thereof.

3D rendering engine 132 can generate digital timing signals 133 for theXY mirror control 134, a digital RGB data stream 170 to analog todigital converter (ADC) 171 and digital sub deflection signals 164.

ADC 171 can generate analog signals representative of the RGBintensities from the 3D rendering engine. Each analog signal can bebuffered by a driver 160 which then can drive a corresponding laser inlaser array 162. Laser array 162 can also include an infrared laserwhich can be used to detect the location of a finger 170 using sensor179. In the alternative, a sensor may be located on the user's hand, forexample on a glove positioned on the user's hand, and the sensor can beused to determine the position of the finger 170 as opposed to thearrangement of FIG. 7 where the sensor 179 detects the position of thefinger 170. In the exemplary embodiment of FIG. 7, the infrared lasercan be modulated by an oscillator 176. When the user interacts with theholographic image, the infrared laser beam can be reflected from theuser's finger 170 and picked up by a plurality of sensors 179 arrangedaround mirror assembly 100, for example. The sense signals can beconditioned by one or more signal conditioners 175, which can includefor example photodiode amplifier(s), or similar apparatus. The outputsfrom the signal conditioners 175 can be then demodulated by one ormultiple demodulators 174 and then filtered by multiple low passfilter(s) 172. ADC 171 can convert the analog signals to a digitalsignal for further processing by the CPU 130, for example. According toan embodiment, each sensor can have its own signal conditioner,demodulator, low pass filter and/or ADC.

Digital sub deflection signals 164 can be comprised of two digitalvectors X_SD and Y_SD, according to an embodiment. A laser beam subdeflection driver can convert X_SD and Y_SD to analog signals that canbe buffered and control laser beam sub deflection modulator 166.

The phase-shift between the signal 176 and the received signal 178 canbe a function of the distance the IR light traveled from laser array 163to each of the sensors 179, among other variables. The demodulation gaincan be a function of the cosine of the phase-shift and thus the digitalresults 177 out of ADC 172 can vary accordingly. In the particularimplementation shown in FIG. 7, results 177 can be comprised of 5digital results, for example, one for each of the sensors 179. Thelocation of where the user interacts with the holographic image can bederived by trilateration based on the plurality of sensor signals,and/or other techniques known in the art.

XY scanner 153 can project one or more beams at any given angle towardthe lens assembly 100. Each micro lens of the top lens 120 can beindividually tuned, in order to provide a desired deflection angle basedon its focal length, depending on the location of an observer. Also, amodulation function can be added to the beam 110 over the entire microlens array (i.e., the entire top lens 120), such that the beam can beactive at desired micro lenses and inactive where light should not beseen from the perspective of the observer (e.g., at predetermineddeflection angles). The desired micro lens at which the beam should beactivated can be determined based on the location of the observer, suchthat the beam(s) can be angled from the micro lens to the eye of theobserver to provide a three-dimensional illusion of the object to bedisplayed. According to this exemplary embodiment, with a plurality ofbeams positioned by one or more micro lenses, the illusion of adisplayed object at a certain point in space can be created.

FIG. 8 shows various object views and associated parameters for anexemplary display apparatus configured to produce an interactivethree-dimensional holographic image, according to one disclosedembodiment. For holographic viewing in a given viewing direction atleast two object views can be needed: a first object view at a firstviewing angle and second object view at a second viewing angle, thefirst object view creating a projection into the left eye, for example,while the second object view creates a projection into the right eye,for example. This can create a plurality of views, each object viewhaving a unique inclination angle θ and azimuth angle ϕ. relative to thesurface of the micro-lens array. A total of R×S views can be generatedwhere S is the maximum number of inclination angles and S is the maximumnumber of azimuth angles.

Optical sensors 179 can be fixed to or removably placed around lensassembly 100 and communicatively coupled to processor unit 130 and/or ahost CPU. When an interactive device, such as a finger or a stylus (afinger 170 is depicted in FIG. 7 for exemplary purposes), interrupts abeam projected above lens assembly 100, the beam can be reflected backand received at one or more of optical sensors 179. Based on themodulation of the reflected beam, for example, which can be previouslyknown by processor unit 130, processor unit 130 can determine where,with respect to the displayed image, the interactive device is located.Using trilateration, based on reflected modulated beams at a pluralityof optical sensors 179, processor unit 130 can determine an accuratelocation of the interactive device (e.g., finger 179), with respect tothe displayed image. Once a location of the interactive device isdetermined, a user can manipulate the displayed image using any knowntouch or multi-touch mechanisms (see FIG. 10 discussed below), withoutdeparting from the scope of the present disclosure.

FIG. 9 illustrates an exemplary feedback system of an exemplary displayapparatus, according to one disclosed embodiment. The feedback system,as depicted in FIG. 9 can be used in order to determine the location ofan object (such as a user's finger 170) and therefore enable the user tointeract with the holographic object. The holographic display apparatuscan have a dedicated touch scan to image the virtual display volume(e.g., the virtual volume into which the holographic object isprojected). The sub deflection function during this touch scan can beadjusted such that the infrared laser beam, for example, exits thecurrently swept micro-lens perpendicularly (i.e. along the focal axis ofthe micro-lens currently being swept).

An object above the micro lens array, according to the depicted example,can cause reflections, which are picked up by a plurality of sensors 179arranged around the micro lens array. The (x,y) coordinate at which thereflection occurs can be determined, as the reflection point coincideswith a particular micro-lens being swept.

FIG. 10 illustrates various exemplary interactions with a holographicdisplayed three-dimensional image according to the present invention. Asshown in FIG. 10, a finger 170 approaching the holographic object mayaccent features of the object, such as intensity at the point where thefinger approaches the holographic object (see the lower right portion ofFIG. 10).

Single finger gestures can be used to rotate and move the object in anydirection. For example, a user's finger 170 approaching the holographicobject from the right can move the object to the left after makingcontact with the objects projected surface to apply a tangential force(see the upper right portion of FIG. 10). Linear motions can haveinertia and the amount of inertia can be a function of how quickly theuser's finger approaches or pushes the holographic object (see the upperleft portion of FIG. 10). Moving the finger parallel to the objectsurface can cause object rotation towards the direction of the fingermovement. Rotation could have inertia such that the object keepsrotating at a rate proportional to the velocity of the finger movement.

Scaling of the holographic object can be accomplished by using at least2 fingers (see the lower left portion of FIG. 10). For example, scalingin the x-direction can be accomplished by touching the left side of theobject with the left index finger and the right side of the object withthe right index finger and then increasing or decreasing the distancebetween the index fingers to scale the holographic object up or down,respectively.

It is noted that processor unit 130 of FIG. 7, and any other processingunits, may include any number of devices or device combinations as knownin the art. These include, for example, general purpose processors,content addressable memory modules, digital signal processors,application-specific integrated circuits, field programmable gatearrays, programmable logic arrays, discrete gate or transistor logic, orother such electronic components.

Furthermore, the steps of a method or algorithm described in connectionwith the embodiments disclosed herein can be embodied directly inhardware, in firmware, in a software module executed by processor unit130 of FIG. 7, or in any practical combination thereof. A softwaremodule can reside in computer-readable storage (not shown), which can berealized as random access memory (RAM), flash memory, read only memory(ROM), erasable programmable ROM (EPROM) memory, electrostaticallyerasable programmable ROM (EEPROM) memory, registers, a hard disk, aremovable disk, a compact disc (CD)-ROM, or any other form of storagemedium known in the art. In this regard, computer-readable storage canbe coupled to processor unit so that the processor unit can readinformation from, and write information to, the computer-readablestorage. In some embodiments, the computer-readable storage can includecache memory for storing temporary variables or other intermediateinformation during execution of instructions by the processor unit. Insome embodiments, the computer-readable storage can include non-volatilememory.

FIG. 11 shows an exemplary detailed 3D rendering engine 132, accordingto an exemplary embodiment. According to the depicted embodiment, 3Dimage memory 300 can include the rendered 3D object. Each object pixel(or vortex) can have the format (R,G,B,X,Y,Z), where R, G and B canprovide the pixel intensities of the red, green and blue pixel colorcomponents, respectively, and X, Y, Z can provide the x, y and zlocation of the pixels, respectively.

A mapping engine 310 can map a given 3D pixel to 2D space based on anazimuth and inclination angle and essentially performs thetransformation: (R,G,B,X,Y,Z)→(R,G,B,X,Y,θ,ϕ), for example. The mappingengine 310 can create a plurality of views based a plurality of viewingangles (θ, ϕ) and store the views in a plurality of view memories321-324.

The pixel sequencing engine 330 can sequence the pixels and viewingangles (θ, ϕ) from the view memories 321-324 based on the current scanlocation. A coordinate transformation engine 340 can map the pluralityof viewing angles (θ, ϕ) in polar form to sub deflection values (X_SD,Y_SD) representative of angles of reflection, for example.

A timing generator 350 can generate the horizontal and verticalsynchronization signals HSYNC and VSYNC, which can be required by themirror driver, and also generate the horizontal and vertical addressesthat can provide a current scan location of the micro lens array. Notethat the 3D rendering engine 132 may be integrated in whole or in partinto a graphics processor unit (GPU) or maybe entirely separate. Imagingmapping engine may generate the views in real time without the need orat least with substantially fewer view memories, according to anembodiment.

FIG. 12 illustrates exemplary computing system 600 that can include oneor more of the embodiments described above. Computing system 600 caninclude one or more of processor 602 and controller 606. Controller 606can include, but is not limited to, transmit section 614, receivesection 607 and display logic 610. Display logic 610 can access RAM 612.In addition, display logic 610 can control transmit logic 614 togenerate an electric field to control the refractive index, and thus aphase shift, at various cells 121, for example. Receive logic 607receives incoming signals from the optical sensor(s) 150/379, in orderto determine the location of an interactive device with respect to thedisplayed image of the object. In some embodiments, controller 606 andprocessor 602 can be integrated into a single application specificintegrated circuit (ASIC).

FIG. 13 illustrates exemplary digital media player 740 that can includea display device 730, the display device 730 including an interactivethree-dimensional holographic display, according to one disclosedembodiment.

FIG. 14 illustrates exemplary personal computer 744 that can include adisplay 730, the display device 730 including an interactivethree-dimensional holographic display, according to one disclosedembodiment.

FIG. 15 illustrates exemplary mobile telephone 736 that can include adisplay device 730, the display device 730 including an interactivethree-dimensional holographic display, according to one disclosedembodiment.

The mobile telephone, media player and personal computer of FIGS. 13, 14and 15, respectively, can provide a three-dimensional viewing andinteracting experience, without the requirement of a reflective mediumor wearing 3D glasses, according to disclosed embodiments. Accordingly,a truly unobtrusive interactive three-dimensional holographic displaycan be provided.

In accordance with the forgoing invention disclosure, it is apparentthat each digital object will be assigned with appropriate physicalproperties as metadata of the digital object, for example, hardness,water content, compressive strength, viscosity etc. A user may scale upor down such physical properties during interaction. When holographic 3Dobject is created, then such physical properties will also be associatedwith the holographic object.

Cameras and proximity sensors installed in the device will analyze handor finger movement pattern around the holographic object and accordinglywill identify the type of applied force, amount of applied force anddetection of applied force on the holographic object.

The device will change the structure and shape of the holographic 3Dobject based on identified force parameters and selected scale. Theshape of the object will be changed accordingly to the laws of physicalobjects. The Device will connect to study material contents andaccordingly projection pattern will be changed to create new or changedholographic 3D shape.

Wearable or embeddable devices fixed in hand/finger and accordinglysoftware can precisely measure the relative finger movement anddirection, this will help to calculate the applied force, type of forceand direction of applied force very precisely. For example, user isusing wearable device (e.g. ring) in thumb and index finger orembeddable device (e-tattoo, or conductive paint) in nails or fingers,so the said devices will measure the relative movement among each otherand also speed and direction of movement, so the paired computingdevices will precisely measure the movement and forces very precisely.

In the conventional art, the physical properties of the selectedmaterial are not considered while changing the shape. With thisinvention, the physical properties of any selected material are defined,e.g. viscosity, cohesiveness, compressive and tensile strength etc.,based holographic object creation. Cameras and sensors installed in thesystem will track user's hand movement type to identify: 1. Type ofapplied force; 2. Direction of applied force; 3. Amount of appliedforce; and 4. Duration of applied force. Then, the shape of theholographic object will be changed. For example, how rotational force onpottery wheel can create different shape from clay may be examined. Auser can use the methods of this invention to create different shapes ofthe holographic object.

With reference to FIGS. 16 and 17, a user may select clay as thepreferred material and also has selected appropriate scale for the claymodel. The simulated force applied by the user will be calculated basedon the selected scale and the input received from sensors mounted on theuser's finger. The holographic system described above will be used tocreate a holographic object. Software will consider the createdholographic object is a simulated clay object. A user can selectdimensions of the holographic object with existing methods known in theart. With this invention, the user may apply a simulated force on theholographic object, and cameras and sensors installed in the device willanalyze the pattern of hand or finger movement, speed of movement andaccordingly will identify type, direction and amount of applied force.

Again, with reference to FIGS. 16 and 17, software has analyzed theamount of applied force, direction of force, and type of applied force,accordingly from the selected scale, and selected simulated materialsoftware has will connect to data stored in the system to calculate thepossible shape, accordingly, the holographic projector will recreate newshape of the object gradually. In this invention, the user willidentifier and visualize how the object is changing based on appliedforce and associated parameters.

FIG. 18 illustrates a computer system 90 used for implementing themethods of the present invention. The computer system 90 includes aprocessor 91, an input device 92 coupled to the processor 91, an outputdevice 93 coupled to the processor 91, and memory devices 94 and 95 eachcoupled to the processor 91. The input device 92 may be, inter alia, akeyboard, a mouse, etc. The output device 93 may be, inter alia, aprinter, a plotter, a computer screen, a magnetic tape, a removable harddisk, a floppy disk, etc. The memory devices 94 and 95 may be, interalia, a hard disk, a floppy disk, a magnetic tape, an optical storagesuch as a compact disc (CD) or a digital video disc (DVD), a dynamicrandom access memory (DRAM), a read-only memory (ROM), etc. The memorydevice 95 includes a computer code 97 which is a computer program thatincludes computer-executable instructions. The computer code 97 includessoftware or program instructions that may implement an algorithm forimplementing methods of the present invention. The processor 91 executesthe computer code 97. The memory device 94 includes input data 96. Theinput data 96 includes input required by the computer code 97. Theoutput device 93 displays output from the computer code 97. Either orboth memory devices 94 and 95 (or one or more additional memory devicesnot shown in FIG. 18) may be used as a computer usable storage medium(or program storage device) having a computer readable program embodiedtherein and/or having other data stored therein, wherein the computerreadable program includes the computer code 97. Generally, a computerprogram product (or, alternatively, an article of manufacture) of thecomputer system 90 may include the computer usable storage medium (orsaid program storage device).

The processor 91 may represent one or more processors. The memory device94 and/or the memory device 95 may represent one or more computerreadable hardware storage devices and/or one or more memories.

Thus the present invention discloses a process for supporting, deployingand/or integrating computer infrastructure, integrating, hosting,maintaining, and deploying computer-readable code into the computersystem 90, wherein the code in combination with the computer system 90is capable of implementing the methods of the present invention.

While FIG. 18 shows the computer system 90 as a particular configurationof hardware and software, any configuration of hardware and software, aswould be known to a person of ordinary skill in the art, may be utilizedfor the purposes stated supra in conjunction with the particularcomputer system 90 of FIG. 9. For example, the memory devices 94 and 95may be portions of a single memory device rather than separate memorydevices.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Although the disclosed embodiments have been fully described withreference to the accompanying drawings, it is to be noted that variouschanges and modifications will become apparent to those skilled in theart. Such changes and modifications are to be understood as beingincluded within the scope of the disclosed embodiments as defined by theappended claims.

What is claimed is:
 1. A method of producing an interactivethree-dimensional holographic image, said method comprising the stepsof: generating, by one or more processors of a computer system, avirtual object, said virtual object being assigned a location within avirtual space of a virtual environment and a movement state at thelocation; assigning physical properties to said virtual object usingmetadata, said physical properties including at least a material of thevirtual object and accompanying material properties; determining aposition of a virtual tool; receiving signals from said virtual tool,wherein the signals include information related to movement of saidvirtual tool relative to said virtual object; calculating an interactiveforce generated on the virtual object by said virtual tool based on saidsignals from said virtual tool and said position of said virtual tool;generating a deformed modified virtual object based on application ofsaid interactive force to said virtual object, wherein the deformedmodified virtual object includes an update to at least one of thelocation and the movement state, and wherein said physical propertiesassigned to said virtual object are accounted for in the deformedmodified virtual object and the update to at least one of the locationand the movement state; and displaying said deformed modified virtualobject as a holographic image within the virtual space of the virtualenvironment.
 2. The method of claim 1, wherein said method furthercomprises: providing additional modification to said virtual objectbased on additional interactive force between said virtual tool and saidvirtual object.
 3. The method of claim 2, wherein said method furthercomprises generating a modified hologram image of said modified virtualobject utilizing said additional modification.
 4. The method of claim 1,further comprising the steps of: determining a force feedback accordingto the position of the virtual tool in relation to the virtual object;sending the force feedback to a user through a haptic interface device;and updating the force feedback according to movement of the virtualtool in real space.
 5. The method of claim 4, wherein said hapticinterface device provides a touch sensation related to an interacting ofsaid user with said virtual object.
 6. The method of claim 4, whereinsaid haptic interface device provides an electro-mechanical linkagewhich can exert a controllable force on a user's hand.
 7. The method ofclaim 4, wherein said haptic interface device generates haptic aspectsof the virtual environment and determines forces to be applied to theuser through said haptic interface.
 8. The method of claim 1, furthercomprising: receiving a selection of a modification mode, saidmodification mode indicating a kind of modification that occurs to saidvirtual object as a result of interaction between said virtual tool andsaid virtual object; wherein said step of generating said deformedmodified virtual object is further based on said selection of saidmodification mode.
 9. The method of claim 1, further comprising thesteps of: determining a position of said virtual tool based on signalsreceived from sensors mounted to a user.
 10. The method of claim 1,further comprising the steps of: determining a position of said virtualtool based on signals received from sensors positioned remote from auser, said sensors detecting a position of said virtual tool.
 11. Acomputer program product comprising: a computer-readable storage device;and a computer-readable program code stored in the computer-readablestorage device, the computer readable program code containinginstructions executable by a processor of a computer system to implementa method of generating a hologram image, the method comprising:generating a virtual object, said virtual object being assigned alocation within a virtual space of a virtual environment and a movementstate at the location; assigning physical properties to said virtualobject using metadata, said physical properties including at least amaterial of the virtual object and accompanying material properties;determining a position of a virtual tool; receiving signals from saidvirtual tool, wherein said signals include information related tomovement of said virtual tool relative to said virtual object;calculating an interactive force generated on the virtual object by saidvirtual tool based on said signals from said virtual tool and saidposition of said virtual tool; generating a deformed modified virtualobject based on application of said interactive force to said virtualobject, wherein the deformed modified virtual object includes an updateto at least one of the location and the movement state, and wherein saidphysical properties assigned to said virtual object are accounted for inthe deformed modified virtual object and the update to at least one ofthe location and the movement state; and displaying said deformedmodified virtual object as a holographic image within the virtual spaceof the virtual environment.
 12. The computer program product of claim11, wherein said method further comprises the step of providingadditional modification to said virtual object based on additioninteractive force between said virtual tool and said virtual object. 13.The computer program product of claim 11, further comprising:determining a force feedback according to the position of the virtualtool in relation to the virtual object; sending the force feedback tothe user through a haptic interface device; and updating the forcefeedback according to movement of the virtual tool in real space. 14.The computer program product of claim 13, wherein said haptic interfacedevice provides a touch sensation related to an interacting of said userwith said virtual object.
 15. The computer program product of claim 13,wherein said haptic interface device provides an electro-mechanicallinkage which can exert a controllable force on a user's hand.
 16. Thecomputer program product of claim 13, wherein said haptic interfacedevice generates haptic aspects of the virtual environment anddetermines the forces to be applied to a user through a hapticinterface.
 17. The computer program product of claim 11, wherein themethod further comprises: receiving a selection of a modification mode,said modification mode indicating a kind of modification that occurs tosaid virtual object as a result of interaction between said virtual tooland said virtual object; wherein said step of generating said deformedmodified virtual object is further based on said selection of saidmodification mode.
 18. A computer system for creating a virtual objectin a virtual environment, the system comprising: a central processingunit (CPU); a memory coupled to said CPU; and a computer readablestorage device coupled to the CPU, the storage device containinginstructions executable by the CPU via the memory to implement a methodof creating a virtual object, the method comprising the steps of:generating a virtual object, said virtual object being assigned alocation within a virtual space of a virtual environment and a movementstate at the location; assigning physical properties to the virtualobject via computer input, said physical properties including at least amaterial of the virtual object and accompanying material properties;determining a position of a virtual tool in the virtual environmentcorresponding to a location of a user in real space; receiving signalsfrom said virtual tool, wherein the signals include information relatedto movement of said virtual tool relative to said virtual object;calculating forces generated on the virtual object by said virtual toolbased on said signals and said position of said virtual tool; modifyingthe shape of said virtual object according to said calculated forcesgenerated by said virtual tool, wherein the deformed modified virtualobject includes an update to at least one of the location and themovement state, and wherein said physical properties assigned to saidvirtual object are accounted for in the deformed modified virtual objectand the update to at least one of the location and the movement state;and displaying said deformed modified virtual object as a holographicimage within the virtual space of the virtual environment.
 19. Thecomputer system of claim 18, wherein said virtual environment comprisesa haptic virtual environment including sensors for determining saidposition of said virtual tool.
 20. The computer system of claim 18,wherein the method further comprises: receiving a selection of amodification mode, said modification mode indicating a kind ofmodification that occurs to said virtual object as a result ofinteraction between said virtual tool and said virtual object; whereinsaid step of modifying the shape of said virtual object is further basedon said selection of said modification mode.